System, apparatus and method for providing improved performance of aggregated/bonded network connections between remote sites

ABSTRACT

A networking system, method, and device is provided for improving network communication performance between client sites at a distance from one another such that would usually require long haul network communication. The networking system includes at least one network bonding/aggregation computer system for bonding or aggregating one or more diverse network connections so as to configure a bonded/aggregated connection that has increased throughput; and at least one network server component implemented at an access point to a high performing network. Data traffic is carried over the bonded/aggregated connection. The network server component automatically terminates the bonded/aggregated connection and passes the data traffic to the network backbone, while providing a managed network path that incorporates both the bonded/aggregated connection and a network path carried over the high performing network, and thereby providing improved performance in long haul network connections.

PRIORITY

This application is a continuation-in-part of U.S. patent applicationSer. No. 13/420,938, filed on Mar. 15, 2012, which is a continuationapplication of U.S. Pat. No. 8,155,158, which are incorporated by thisreference.

FIELD OF THE INVENTION

The present invention relates generally to network communications and,in particular, to aggregating or bonding communications links so as toimprove network performance such as Cable, DSL, T1, or Metro Ethernet,so as to improve network performance or quality of services in regardsto a variety of different networks including wired and wirelessnetworks, and including Wide Area Networks (“WAN”).

BACKGROUND

While the capacity of network connections has increased since theintroduction of dial up, high speed connectivity is not ubiquitous inall regions. Also, bandwidth is not an unlimited resource and there is aneed for solutions that improve the utilization bandwidth and also thataddress network performance issues.

Various solutions exist for improving network performance such as loadbalancing, bonding of links to increase throughput, as well asaggregation of links. In regards to bonding/aggregation variousdifferent technologies exist that allow two or more diverse links (whichin this disclosure refers to links associated with different types ofnetworks and/or different network carriers) are associated with oneanother for carrying network traffic (such as a set of packets) acrosssuch associated links to improve network performance in relation forsuch packets.

Examples of such technologies include load balancing, WAN optimization,or ANA™ technology of TELoIP™, as well as WAN aggregation technologies.Many of such technologies for improving network performance are used toincrease network performance between two or more locations (for exampleLocation A, Location B, Location N or the “Locations”), wherebonding/aggregation of links is provided at one or more of suchlocations. While the bonded/aggregated links provide significant networkperformance improvement over the connections available to carry networktraffic for example from Location A to an access point to the backboneof a network (whether an Internet access point, or access point toanother data network such as a private data network or high performancewireless network) (“network backbone”), the bonded/aggregated links aregenerally slower than the network backbone.

Prior art technologies including bonding/aggregation generally result inwhat is often referred to as “long haul” bonding/aggregation, whichmeans that the bonded/aggregated links are maintained for example fromLocation A and Location B, including across the network backbone, whichin many cases results in network impedance. As a result, whilebonding/aggregation provides improved network performance for examplefrom Location A to the network backbone, network performance across theentire network path for example from Location A to Location B, may beless than optimal because the technology in this case does not take fulladvantage of the network performance of the network backbone.

There is a need for a system and method that addresses at least some ofthese problems.

SUMMARY OF THE INVENTION

In one aspect, a networking system is provided for improving networkcommunication performance between at least a first client site and asecond client site, where the first client site and the second clientsite are at a distance from one another that is such that would usuallyrequire long haul network communication, comprising: (A) at least onenetwork bonding/aggregation computer system that includes: (i) at leastone client site network component that is implemented at least at afirst client site, the client site network component bonding oraggregating one or more diverse network connections so as to configure abonded/aggregated connection that has increased throughput; and (ii) atleast one network server component, configured to interoperate with theclient site network component, the network server component including aserver/concentrator that is implemented at an access point to a highperforming network;

wherein the client site network component and the network servercomponent are configured to interoperate so as to create and maintain anetwork overlay for managing network communications between the at leastfirst client site and the access point, wherein between the client sitenetwork component and the network server component data traffic iscarried over the bonded/aggregated connection and between the accesspoint and the second client site the network server componentautomatically terminates the bonded/aggregated connection and passes thedata traffic to the associated network backbone, while maintainingmanagement of data traffic so as to provide a managed network path thatincorporates both at least the bonded/aggregated connection and at leastone network path carried over the high performing network, and therebyproviding improved performance in long haul network connections betweenthe first client site and the second client site.

In another aspect, the first client site and the second client site areat a distance from one another such that data traffic over abonded/aggregated connection between the first client site and thesecond client site is subject to long haul effects.

In another aspect, the managed network path is maintained between atleast a first client site and a second client site without routing ofnetwork communications through a central server that increases long hauleffects.

In another aspect, the plurality of network server components areimplemented in a geographic area so as to provide a Point-of-Presence(PoP), which are made available to proximate client site networkcomponents.

In a still another aspect, the two or more Points-of-Presence areaccessible to the at least one client site network component, and theclient site network component: collects network performance information;and initiates the configuration of the network overlay to include one ormore network server components so as to improve network communicationperformance.

In another aspect, a computer implemented method for improving networkcommunication performance between at least two sites, is provided wherethe two sites are at a distance from one another that is such that wouldusually require long haul network communication, comprising: (A) using aclient site network component associated with a first client site toconnect to a proximal network server component, the network servercomponent being connected to an access point to a high performingnetwork, forming thereby a network overlay that provides a bonded oraggregated connections for carrying data packets; (B) the network servercomponent terminating the bonded or aggregated connection; and (C) thenetwork server component transferring the data packets to the highperforming network, for delivery to a second client site, whilemaintaining management of data traffic so as to provide a managednetwork path that incorporates at least the bonded/aggregated connectionand at least one network path carried over the high performing network,thereby reducing long haul effects.

In this respect, before explaining at least one embodiment of theinvention in detail, it is to be understood that the invention is notlimited in its application to the details of construction and to thearrangements of the components set forth in the following description orillustrated in the drawings. The invention is capable of otherembodiments and of being practiced and carried out in various ways.Also, it is to be understood that the phraseology and terminologyemployed herein are for the purpose of description and should not beregarded as limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

Examples of embodiments of the invention will now be described ingreater detail with reference to the accompanying drawings, in which:

FIG. 1 a illustrates a prior art network configuration that includes abonded/aggregated network connection. FIG. 1 a illustrates the problemof long haul aggregation/bonding.

FIG. 1 b also illustrates a prior art network configuration thatincludes central management of bonded/aggregated network connections.

FIG. 2 a shows a network solution in accordance with the presentinvention, with bonding/aggregation implemented at both Site A and SiteB, while minimizing long haul effects based on the technology of thepresent invention.

FIG. 2 b shows another network solution in accordance with the presentinvention, in which bonded/aggregated network service exists at Site Abut not at Site B.

FIG. 2 c shows a still other network solution in accordance with thepresent invention, in which bonding/aggregation is implemented asbetween Site A, Site B, and Site C.

FIG. 2 d shows a further implementation of the network architecture ofthe present invention, in which a plurality of servers/concentrators areimplemented as part of a Point-of-Presence.

FIG. 3 is a block diagram of a communication device incorporating aparticular embodiment of the invention, demonstrating the device as anaggregation means on the client/CPE side of a network connection.

FIG. 4 is a block diagram of a communication device incorporating aparticular embodiment of the invention, demonstrating the device as anaggregation means on the server/concentrator side of a networkconnection.

FIG. 5 is a block diagram of a communication network incorporating aparticular embodiment of the invention, demonstrating the device as anaggregation means on both the client/CPE side and server/concentratorside of a network connection.

FIG. 6 is a flow diagram of a method of providing redundancy andincreased throughput through a plurality of network connections in anaggregated network connection.

FIG. 7 a illustrates a prior art network architecture where long hauleffects apply, and presents network performance based on download speed.

FIG. 7 b illustrates, in similar network conditions as in FIG. 7 a butimplementing the present invention in order to reduce long haulbonding/aggregation, improved network performance based on fasterdownload speed.

DETAILED DESCRIPTION

In one aspect, a novel and innovative system, network architecture andnetworking method is provided.

In another aspect of the present invention, a network solution isprovided for improving network communication performance between atleast two sites, where the two sites are at a distance from one anotherthat is such that would usually require long haul network communication.The network solutions includes at least one network bonding/aggregationsystem that includes (A) at least one first network component that isimplemented at a first service site, the first network component beingconfigured to bond or aggregate one or more diverse network connectionsso as to configure a bonded/aggregated connection that has increasedthroughput; and (B) a second network component, configured tointeroperate with the first network component, the second networkcomponent including a server/concentrator (also referred to as networkserver component) that is implemented at an access point to a highperforming network backbone.

In one aspect, the first network component, may be implemented usingwhat is called in this disclosure a “CPE” or customer premises equipment(also referred to as client site network component). The CPE and aserver/concentrator component (more fully described below) interoperateto configure the bonded/aggregated connections in order to provideimprove network connections at a site associated with the CPE.

In one aspect of the present invention, the server/concentrator isimplemented at an access point, with access to the network backbone soas to avoid long-haul bonded/aggregated network communications. As setout in the Example in Operation cited below, network architectures thatinvolve long-haul bonded/aggregated network communication result in lessthan optimal performance, thereby minimizing the advantages of thebonding/aggregation technology. In other words, while thebonding/aggregation technology may improve service to Site A associatedwith for example a CPE (or equivalent), based on bonding/aggregationbetween the CPE and an associated server/concentrator (or equivalent),overall performance may be less than desired and in fact may be lessthan what would be available without bonding/aggregation because of thelong haul effects of carrying the bonded/aggregated from Site A, to atleast Site B. These long haul effects will present wherever Site A andat least Site B are at a substantial distance from one another.

The Example in Operation below illustrates the decrease in performancethat results from the long haul effects.

FIG. 1 a illustrates the problem of long haul aggregation/bondinggenerally. In the prior art bonded/aggregated network communicationshown in FIG. 1 a, packets are carried over the Internet through anextension of the bonded/aggregated connection across the Internet (102),rather than the high performing Internet. The bonded/aggregatedconnection, across a distance that is subject to long haul effects, willnot perform as well as the Internet, thereby providing less than idealperformance.

The Example in Operation reflects another problem with prior artbonding/aggregation solutions, namely that they generally requirecontrol or management by a central server. Depending on the location ofthe central server, this can result in multiplying of the long hauleffects because traffic between a Site A and Site B may need to also betransferred to a Site C that is associated with the central server. Thisaspect of the invention is illustrated for example in FIG. 1 b. Centralserver (104) manages network communications, and in fact routes networkcommunications between Site A and Site C. To the extent that thedistance between central server (104) is substantial from either of SiteA or Site C, long haul effects will present. If central server (104) isat a substantial distance from each of Site A and Site C, then therewill be a multiplying of the long haul effects, as network traffic willpass from Site A to the central server (104) to Site C, and from Site Cto the central server (104) to Site A.

As illustrated in the Example of Operation, long haul effects have anegative impact on speed (slowing traffic) and also on latency.Conversely, the present invention provides significant improvements inregards to both speed and latency.

The present invention provides a novel and innovative network solution,including a networking system and architecture and associated networkingmethod, that addresses the aforesaid long haul effects that have anegative effect on performance.

As shown in FIG. 2 a, in one aspect of the invention, theserver/concentrator side of a bonding/aggregation network solution forSite A (120 a) is implemented such that (A) the location of theserver/concentrator is implemented with access to the network backboneof Internet (112), and (B) the server/concentrator (110 a) includesfunctionality for (i) receiving packets by means of thebonded/aggregated connection (116 a), (ii) interrupting thebonded/aggregated connection (116 a) using an interruptor (118), and(iii) directing the packets (114) to the Internet (112) for delivery toa Site B (120 b). If Site B also has bonded/aggregated network service,then the packets are delivered to a Site B side server/concentrator (110b). Server/concentrator (110 b) established a further bonded/aggregatedconnection (116 b) and directs the packets (114) via thebonded/aggregated connection (116 b) to a CPE(B) (124 b) at Site B.

FIG. 2 b illustrates a configuration where bonded/aggregated networkservice exists at Site A but not at Site B.

A skilled reader will understand that more than two sites are possible,where the network system of the present invention improves networkperformance for network communications between for example Site A, SiteB, and Site C where one or more sites will include bonded/aggregatedservice. In one implementation of the invention, as shown in FIG. 2 c,bonded/aggregated service is present for each of Site A, Site B and SiteC. FIG. 2 c illustrates one possible implementation of the invention,where the network system is based on a distributed network architecturewhere server/concentrators (110 a) (110 b) (110 c) and correspondingCPEs (124 a) (124 b) (124 c) are configured to provide improved networkcommunications, including interruption of network communications at thenetwork backbone so as to reduce long haul effects, dynamically and on apeer to peer basis without the need for a persistent central manager. Inone implementation, each of the network components of the network systemincluded functionality to operate on a peer to peer basis.

A CPE (124) initiates network communications on a bonded/aggregatedbasis, cooperating with a server/concentrator (110), with packetsdestined for a remote location. Each server/concentrator (110) receivesdynamic updates including a location and identifier associated withother server/concentrators (110). Packets are dynamically sent to aserver/concentrator (110) at the remote location, if available, and fromthe server/concentrator (110) at the remote location to its CPE (124).The CPEs (124) and their server/concentrators (110) use bi-directionalcontrol of network communications to establish a network overlay toprovide improved network performance. The network overlay for exampleprovides desirable quality of service despite underlying networkconditions that may otherwise resulted in a decrease in networkperformance.

In accordance with the present invention, the network system of thepresent invention establishes and manages two or more network overlays.Referring for example to FIG. 2 a a first network overlay (126) isestablished between the CPE(A) (124 a) and server/concentrator (110 a);then, communications are transferred over the Internet (112) without anetwork overlay; then, a second network overlay (129) is establishedbetween server/concentrator (110 b) and CPE(B) (124 b). As a result, IPtransport is provided between Site A and Site B where this will providebetter performance than the aggregated/bonded network connections.Bonding/aggregation in effect is distributed across the locations,rather than attempting to span the distance between the locations withend to end bonding/aggregation.

The present invention therefore provided distributedbonding/aggregation. The present invention also provides a networksystem that automatically provides distributed bonding/aggregation in away that bonding/aggregation is proximal, and beyond proximalconnections IP transport is used, with proximal bonded/aggregatedconnections and fast Internet being used as part of end-to-end improvedservice.

In another aspect of the invention, and as shown in FIG. 2 d, one ormore server/concentrators can be implemented at a physical location, aspart of a Point-of Presence (PoP) (130). In one aspect, in the contextof the present invention, a PoP (130) can define a relatively highconcentration of servers/concentrators within an area. In anotheraspect, a plurality of PoPs (130) may be available in a geographiclocation. A plurality of PoPs (130) may be established based on networktopology or service requirements in a given area.

In one aspect, each PoP (130) has one or more network backboneconnections (132), because in some locations different network backbonesmay be available. The PoP (130) may be implemented so that itdynamically interoperates with surrounding networks. The PoP (130) is acollection of network components, established at the periphery of thenetwork backbone (112), associated with a plurality of networks, andcumulatively providing network communication service to a one or moreclients in a defined geographic area. In one possible implementation,the server/concentrators (110) located within the PoP (130) functions asa network access server for connecting to the Internet (112). Thenetwork access server (110) acts as the access point to the Internet(112) for a plurality of CPE devices (124) that are connected to the PoP(130). The servers/concentrators (110) may be configured to communicatewith one another to share information regarding network conditions.Servers/concentrators (110) provide connectivity to CPEs (124) and mayalso run a networking protocol such as BGP to route servers and othernetwork backbone connections (112).

In one aspect, servers/concentrators (110) are configured to detectchanges in their network environment.

The CPE (124) may be configured to collect information from networkcomponents in its vicinity including from one or more available PoPs(130) and their servers/concentrators (110). The CPE (124) for exampleconnects to a closest available server/concentrator (124), implementedas part of a PoP (130), and thereby having access to a connection to thenetwork backbone (112). Whether the connection to the network backbone(112) is direct or indirect, the network connections are established soas to minimize long haul effects.

In one implementation, each CPE (124) wanting to establish a connectiondynamically advertises its IP address, and receives replies fromassociated servers/concentrators (110) along with their current networkperformance information. The CPE (124) initiates a bonded/aggregatedconnection with a server/concentrator (110) that is both proximal (tominimize long haul effects between the CPE (124) to the network backbone(112)), and also that based on network conditions relevant to theparticular server/concentrator, is performing well.

In one implementation, a network device is deployed that bonds oraggregate multiple, diverse links. The network device may be WANaggregator or a link aggregator.

Once the network overlay is established, various other networkoptimization and quality of services (“QOS”) techniques may be applied.

One or more CPEs and one or more concentrators can create variousdifferent network configurations that improve network performance inrelation to network communications between them. The CPEs andconcentrators are designed to be self-configuring, and to interoperatewith one another to manage traffic in a more effective way.

“Proximal” means a distance such that based on relevant networkconditions, long haul network communication and associated effects areavoided. The distance between the CPE and the server/concentrator isproximal, thereby enabling good network service.

A skilled reader will also understand that in order to take advantage ofthe network architecture of the present invention, theserver/concentrator (110) should be located at an access point to thenetwork backbone (112) or in some other way to minimize the long hauleffect, for example, by the server/concentrator being located proximalto an access point so as to further avoid long haul networkcommunication.

In another aspect, the bonded/aggregated connection at Site A and thebonded/aggregated connection at Site B may be different, in the sensethat each may include different types of network connections and thatmay be associated with different carriers. In one aspect of theinvention, the network overlay provided operates notwithstanding suchdiversity.

The more sites that have the CPE/concentrators associated with them thebetter network performance between them. Representative performancedetails are included below.

The network backbone (112) could be any high performance networkincluding for example a private WAN, the Internet, or an MPLS network.

Network Overlay

In one aspect of the invention, one or more network overlays areestablished in accordance with the present invention, thereby in oneaspect providing a multi-POP network that exploits multiple points ofpresence so as to provide a persistent, configurable/reconfigurablenetwork configuration that provides substantial network performanceimprovements over prior art methods.

In one aspect of the invention, the CPEs/concentrators may monitornetwork performance, including in the areas proximate to their position,and may reconfigure the network overlay dynamically, across multiplelocations (including multiple PoPs) based on changes in networkperformance while providing continuity of service.

In one aspect, the network components of the present invention areintelligent, and iteratively collect network performance information.Significantly, in one aspect each CPE is able to direct associatedconcentrator(s) and any CPE to in aggregate re-configure the networkoverlay.

Significantly, in the network overlay created by the present inventionmanagement of the network may be centralized or decentralized, dependingon the configuration that provides the best overall performance. This isin contrast to prior art solutions that generally require centralmanagement for example of termination of connection which results intraffic being carrier over bonded/aggregated connection that involvelong haul transmission that fail to take advantage of network paths thatmay provide inherently better performance than the bonded/aggregatedconnection paths.

In one aspect, decentralized managed is made possible by peer-to-peerfunctionality implemented to the network components of the presentinvention.

In another aspect of the invention, a plurality of servers/concentratorsmay be established in multiple locations covering a plurality ofdifferent access points. Each server/concentrator may be used formultiple clients associated with different CPEs to improve networkperformance for such multiple clients by providing termination of theirbonded/aggregated connection and transfer of communications to thenetwork backbone. The network solution of the present inventiontherefore may include multiple Points-of-Presence, distributedgeographically including for example in areas requiring network service,and through the network architecture of the present invention bridginggeographically disparate areas with improved network communicationtherebetween.

Additional Implementation Detail

As previously stated, the present invention may be implemented inconnection with any technology for bonding or aggregating links, andthereby reduce long haul effects.

What follows is additional detail regarding link aggregation, which isone form of bonding/aggregation that may be used as part of the overallnetwork solution and network architecture disclosed in this invention.

In one aspect of the invention, the system, method and networkarchitecture may be implemented such that the aggregated/bonded networkconnections described are implemented using the link aggregationtechnology described in U.S. Pat. No. 8,155,158. What follows is furtherdiscussion of possible embodiments of the CPE and theserver/concentrator (or concentrator) components previously described,emphasizing their creation and management of the bonded/aggregatedconnections between them, which in the network configuration of thepresent invention form a part of the overall network overlay thatincorporates the one or more portions that are carried over the networkbackbone.

Diverse network connections may be aggregated into a virtual (logical)connections that provide higher throughput as well as independence ofthe network characteristics of the constituent (physical) network.Aggregation may be performed to a given CPE.

For instance, in one example of the implementation of the presentinvention a Metro Ethernet 10 Mbps (E10) link and a T1 (DS1) link areaggregated in accordance with the invention as described below, in orderto provide higher fault tolerance and improved access speeds. Theaggregation of diverse carriers in accordance may extend to anybroadband network connection including Digital Subscriber Line (DSL)communications links, Data over Cable Service Interface Specification(DOCSIS), Integrated Services Digital Network, Multi protocol LabelSwitching, Asynchronous Transfer Mode (ATM), and Ethernet, etc. Thenetwork connections may also include a WAN.

According to one aspect of the invention, an apparatus is provided formanaging transfer of communication traffic over diverse networkconnections aggregated into a single autonomous connection, independentof the various underlying network connections. The apparatus may includea network aggregation device and an aggregation engine. The networkaggregation device may be adapted to configure a plurality of networkconnections, which transfers communication traffic between a furthernetwork connection and the plurality of network connections, as anaggregated group for providing a transfer rate on the furthercommunication link, and to allocate to the aggregate group a rate oftransfer equal to the total available transfer rate of the underlyingnetworks. The aggregation engine may be adapted to manage thedistribution of communication traffic received both to and from aplurality of network connections, establishing newly formed aggregatednetwork connections. The aggregation engine may be implemented insoftware for execution by a processor, or in hardware, in a manner thatis known to those skilled in the art.

In accordance with this aspect of the present invention, a plurality ofdiverse network connections may be aggregated to create an aggregatednetwork connection. The diversity of the network connections may be aresult of diversity in provider networks due to the usage of differentequipment vendors, network architectures/topologies, internal routingprotocols, transmission media and even routing policies. Thesediversities may lead to different network connections with differentlatencies and/or jitter on the network connection. Also, variationwithin transmission paths in a single provider network may lead tolatency and/or jitter variations within a network connection.

Latency and jitter typically affect all data communication across thenetwork connection. Latency, as is known to those skilled in the art, isthe round-trip time for a transmission occurring end-to-end on a networkconnection. Jitter, as is known to those skilled in the art, is thevariance in latency on a network connection for the same data flow. Highlatency and jitter typically have a direct and significant impact onapplication performance and bandwidth. Applications such as VOIP, andvideo delivery are typically highly sensitive to jitter and latencyincreases and can degrade as they increase.

Transparent aggregation of a plurality of network connections in anaggregated network connection requires the management of datatransmitted over the aggregated connection by the aggregation engine andreceived from the aggregation traffic termination engine. In one aspectof the present invention, transparent aggregation does not require anyconfiguration by a network provider. The aggregation engine and theaggregation traffic termination engine may manage data transmission suchthat the variable path speeds and latencies on the plurality of networkconnections do not affect the application data transmitted over theaggregated network connection. The network aggregation engine and theaggregation traffic termination engine may handle sequencing andsegmentation of the data transmitted through the aggregated connectionto transparently deliver application data through the aggregatedconnection with minimal possible delay while ensuring the ordereddelivery of application data.

In one aspect of the invention, the network aggregation engine providesa newly aggregated network connection with a capacity equal to the sumof the configured maximum throughput of the network connections.

The aggregation engine and an aggregation traffic termination engine(further explained below) handle the segmentation of packets as requiredin confirmation with architectural specifications such as MaximumSegment Size (MSS) and Maximum Transmission Unit of the underlyingnetwork connections. The network aggregation device is operable tohandle assignment of sequence identifiers to packets transmitted throughthe aggregated network connection for the purpose of maintaining theordering of transmitted data units over the aggregated networkconnection.

In a further aspect of the invention, the network connection deviceincludes or is linked to a connection termination device, and aplurality of fixed or hot swappable transceivers for transmittingcommunication traffic on respective sets of network connections, for thepurpose of configuring a plurality of network connections as anaggregated connection or the management of multiple aggregated networkconnections and providing access to the aggregated network connectionfor any network communications traversing the device.

In the present specification, routing protocols or route selectionmechanisms described are intended only to provide an example but not tolimit the scope of the invention in any manner.

FIG. 3 is a block diagram of a communication device incorporating aparticular embodiment of the invention, demonstrating the device actingas a client.

As shown in FIG. 3, the network element/network aggregation device (alsoreferred to in this disclosure simply as the “device” or the “networkaggregation device”) 23 includes (in this particular embodiment shownfor illustration) a network connection termination module 25 thatincludes representative transceiver interfaces 14, 15 and 16. Eachtransceiver interface 14, 15 and 16 represents an interface to aphysical communication medium through which communications may beestablished to network connections.

A possible implementation of the network aggregation device may use asingle or multiple chassis with slots for multiple network connectiontermination modules and multiple network aggregation engine modules. Themultiple network connection termination modules may be grouped byprotocol specific or medium specific transceiver/interfaces.

The network aggregation engine 11 may handle the configuration of thenetwork aggregation device and all related interactions with externalinputs. A device configuration store 24 may provide persistent datastorage for device configuration information such as a networkaggregation policy.

The network aggregation engine 11 may handle queries from externalsources, such as configuration parameters a network management protocolsuch as Simple Network Management Protocol, for example. The interface10 may be a protocol agent and may provide for communication with aNetwork Management System (NMS) or operator system for configuration ofthe aggregation engine by the definition of an aggregation policy.Control and management information may be transferred between thenetwork aggregation device 23 and the NMS or operator system through theinterface 10 via any available or specifically designated networkconnection 19, 20, 21 and 17 through any transceiver interface 14, 15and 16.

In accordance with an aspect of the present invention, multiple networkconnections may be combined to form an aggregated network connection 22,as disclosed in further detail herein. Each individual networkconnection may be configured with a maximum communication traffic rate,which could be expressed as a bit rate in bits per second.

The network aggregation engine 11 may be implemented in software forexecution by a processor in the network aggregation device 23, or inhardware such as by means of a Field Programmable Gate Array (FPGA) orother integrated circuit, or some combination thereof. The networkaggregation engine 11 may be implemented in a distributed manner bydistributing aggregation engine intelligence to the network connectiontermination module 25, in a manner that is known.

The network aggregation engine 11 may receive traffic from clientnetwork connection device 18 through a network connection 17 providedthrough a transceiver interface 16. The client network connection device18 may be any device including, without limitation, a router, switch, ormedia converter that is capable of providing termination for a single ormultiple client nodes, where nodes are any devices capable of connectingto a network irrespective of protocol or interface specificity. Invarious embodiments, traffic may be received over multiple networkconnections through a single or multiple transceiver interfaces. Thenetwork aggregation engine 11 may accept all traffic from the clientnetwork connection, may provide encapsulation and segmentation servicesfor the traffic for transmission through the aggregated networkconnection 22, and may transmit it over any of the network connections19, 20 and 21 through any of the transceiver interfaces 14, 15 and 16.The network aggregation engine 11 may handle segmentation in a mannerthat avoids the fragmentation of aggregated communication trafficreceived through the client network connection device 18, whentransmission occurs over the aggregated network connection 22 throughany of the network connections 19, 20 and 21, by ensuring that thelength of a packet/frame transmitted over any of the network connections19, 20 and 21 is less than or equal to the configured or detected framelength for the respective connections in the aggregated networkconnection 22.

The network aggregation engine 11 may poll the state of networkconnections 19, 20 and 21, for example as per configured intervalsstored in the device configuration store 24, to ensure that all networkconnections configured in an aggregated group are within configuredacceptable tolerances. If a network connection 19, 20, and 21 exceedsacceptable tolerance values for any of the polled parameters, thenetwork aggregation engine 11 may remove the network connection 19, 20,and 21 from within the aggregated network connection 22 without removingit from the polled network connections list. By leaving the removednetwork connection 19, 20, and 21 in the polled network connection list,the network aggregation engine 11 may aggregate the network connectioninto the aggregated network connection 22 once it has come back withinacceptable tolerance values. This may ensure that a network connectionmay change states between residing in an aggregated network connection22 or not, without the intervention of an external system or input. Thenetwork aggregation engine 11 may handle notifications to all end pointsconfigured within the device configuration store 24 with internal eventssuch as changes in network connection state, threshold violations onconfigured thresholds for any number of configurable variables for anyobject within or connected to the network aggregation device 23. Thenetwork aggregation engine 12 may also handle events such as changes inthe state of a network connection 19, 20, and 21 included in theaggregated connection, changes in latency of a network connectionincluded in the aggregated network connection 22, scheduling changes,event logging, and other events.

FIG. 4 is a block diagram of a communication device incorporating aparticular embodiment of the invention, demonstrating the device actingas a server/concentrator.

The network aggregation engine 11 may provide access to a networkaggregation policy database 36 which stores configuration informationrelated to the various aggregated network connections that terminate onthe aggregated network connection device 28. The network aggregationtermination device 28 may be implemented in such a manner that eachaggregated network connection defined in the network aggregation policydatabase 36 is handled by its own virtual instance, the use of whichenables termination of each aggregated network connection from multiplecustomer premises equipment (CPE).

FIG. 5 is a block diagram of a communication network incorporating aparticular embodiment of the invention, demonstrating the function ofthe device acting as a client/CPE and server/concentrator.

In accordance with a particular embodiment of the invention, aggregatednetwork connections 70, 71 and 72 may be built by network aggregationdevices 63, 64 and 65, which terminate to a single aggregated networkconnection termination device 61 through network connections 66 and 68as their endpoint. The aggregated network connection termination device61 may access external communications networks through networkconnections 66 and 68 to access external/remote network resource 69.Access to external communications networks may be provided by theaggregated network connection termination device 61 by using eithernetwork connection 66 or 68 through the use of a routing protocol, suchas Border Gateway Protocol (BGP), Open Shortest Path (OSPF), or throughthe use of simpler mechanisms such as load sharing over multiple staticroutes within the communication network 74 that acts as the validnext-hop for the aggregated network connection termination device 61.

Aggregated network connections 70, 71 and 72 may provide access toclient network nodes 67 connected to the network aggregation devices 63,64 and 65 through the aggregated network connections 70, 71 and 72 tocommunications networks 74 accessible by the aggregated networkconnection termination device 61.

A client network node 67 may request data provided by an external/remotenetwork resource 69 accessible through a communication network 74. Thisrequest for the external/remote network resource may be routed over thenetwork connection 73 providing access from the client network node 67over the aggregated network connection 70 to its end point which is theaggregated network connection termination device 61. This may be donethrough the communication network 74 through the network connection 66into the aggregated network connection termination device 61. Any datasent by the external/remote network resource 69 may be routed backthrough the aggregated network connection termination device.

A particular embodiment of this invention may use the Internet as thecommunication network 74 referenced in FIG. 5. However, thecommunication network 74 may alternatively be built by multiplesub-networks created through the use of multiple network aggregationdevices 63, 64 and 65 with aggregated network connection terminationdevice 61 end points through multiple network connections 66 and 68.

A further aspect of the invention relates to the provisioning of highavailability over the aggregated network connection by the networkaggregation engine 11. FIG. 6 illustrates a method of providingredundancy and increased throughput through a plurality of networkconnections in an aggregated network connection. The method 90 may beginwith a step of configuring a plurality of network connections 91 throughthe creation of a network aggregation policy to form 92 the aggregatednetwork connection. The aggregated network connection may be initializedas per the network aggregation policy. Control connections may becreated 93 for the plurality of network connections configured as partof the aggregated connection to allow the aggregation engine 11 tomanage the membership of a network connection within the aggregatedconnection. The network aggregation engine 11 may accept packets fortransmission 94 over the aggregated network connection 22. The networkaggregation engine 11 may choose a network connection 95 among the groupof network connections configured 91 in the aggregate in the storedaggregation policy for transmission of the current packet beingtransmitted. The choice of network connection for transmission of thecurrent packet may be specified within the aggregation policy and maytake into account data provided by the control connection built at 94.

According to one embodiment of the invention, a non-responsive networkconnection may be easily detected when using latency and packet loss asa measure. The mechanism for detecting 96 and adapting to 97 the networkconnection change within an aggregated network connection may beimplemented within the data transmission routine in the aggregationengine 11 or as a separate process in parallel to the transmissionroutine in the aggregation engine 11 to allow for further flexibility inprovisioning redundancy within the aggregated network connection.

Since this may occur on a per packet basis as opposed to on a per streambasis, a single non-responsive network connection may not affect theaggregated network connection and may allow data transmission tocontinue regardless of the individual states of network connections solong as a single network connection within the aggregated networkconnection is available for data transmission.

Example In Operation

In one possible implementation of the present invention, 3 locations areprovided namely Site A, Site B, and Site C, and Site D. FIGS. 7 a and 7b illustrate network performance as discussed herein. FIG. 7 aillustrates performance with long haul effects. FIG. 7 b illustratesperformance with reduction of long haul effects, based on the presentinvention in network conditions otherwise similar to those on which FIG.7 a is based.

FIG. 7 b shows an improvement in performance over FIG. 7 a, based onreduction of long haul effects in relatively long distance networkcommunications are implemented using the network architecture.

The present invention therefore provides improved network performancerelative to speed. A skilled reader will appreciate that the improvementin performance shown in significant. Other aspects of networkperformance are also improved, based on the present invention, forexample latency.

Advantages and Use Case

The present invention significantly improves network performance betweendisparate locations by leveraging network bonding/aggregationtechnology, but by implementing a system, method and networkconfiguration that provides intervening network components disposedadjacent to access points so as to manage traffic between two or moresites such that bonded/aggregated connections are terminated and trafficis directed to a network backbone, and optionally passed to one or morefurther bonded/aggregated connections associated with a remoteadditional site.

The network solutions of the present invention is flexible, responsive,scalable and easy to implement. New sites, optionally having their ownCPE/concentrator can be easily added, and the network solution supportsvarious types of multi-point network communications, and various networkperformance improvement strategies including various QoS techniques.

The network solution is easily updated with new programming or logicthat is automatically distributed on a peer to peer basis based on theinteroperation of network components that is inherent to their design,as previously described.

Network performance is significantly improved over prior art solutionsas illustrated in the Example In Operation provided above.

1. A networking system is provided for improving network communicationperformance between at least a first client site and a second clientsite, where the first client site and the second client site are at adistance from one another that is such that would usually require longhaul network communication, comprising: (a) at least one networkbonding/aggregation computer system that includes: (i) at least oneclient site network component that is implemented at least at a firstclient site, the client site network component bonding or aggregatingone or more diverse network connections so as to configure abonded/aggregated connection that has increased throughput; and (ii) atleast one network server component, configured to interoperate with theclient site network component, the network server component including aserver/concentrator that is implemented at an access point to a highperforming network; wherein the client site network component and thenetwork server component are configured to interoperate so as to createand maintain a network overlay for managing network communicationsbetween the at least first client site and the access point, whereinbetween the client site network component and the network servercomponent data traffic is carried over the bonded/aggregated connectionand between the access point and the second client site the networkserver component automatically terminates the bonded/aggregatedconnection and passes the data traffic to the associated networkbackbone, while maintaining management of data traffic so as to providea managed network path that incorporates both at least thebonded/aggregated connection and at least one network path carried overthe high performing network, and thereby providing improved performancein long haul network connections between the first client site and thesecond client site.
 2. The network system of claim 1, wherein the firstclient site and the second client site are at a distance from oneanother such that data traffic over a bonded/aggregated connectionbetween the first client site and the second client site is subject tolong haul effects.
 3. The network system of claim 1, wherein the managednetwork path is maintained between at least a first client site and asecond client site without routing of network communications through acentral server that increases long haul effects.
 4. The network systemof claim 1, wherein one or more client site network components and oneor more associated network server components include peer-to-peerprogramming and based on such peer-to-peer programming operate on a peerto peer basis.
 5. The network system of claim 1, wherein the networkserver component is disposed at a distance from an access point thatdoes not result in long haul effects between the network servercomponent and the access point.
 6. The network system of claim 1,wherein a plurality of network server components are implemented in ageographic area so as to provide a Point-of-Presence (PoP), which aremade available to proximate client site network components.
 7. Thenetwork system of claim 6, wherein two or more Points-of-Presence areaccessible to the at least one client site network component, and theclient site network component: (a) collects network performanceinformation; and (b) initiates the configuration of the network overlayto include one or more network server components so as to improvenetwork communication performance.
 8. The networking system of claim 1,wherein each network server component is accessible to a plurality ofclient site network components, each client site network component beingassociated with a client site.
 9. The networking system of claim 6,comprising a network of Points-of-Presence, distributed geographicallyso as to serve a plurality of client locations each associated with atleast one client site network component.
 10. The networking system ofclaim 1, comprising: (a) a client site network component at each of thefirst client site, and at the second client site; (b) a network servercomponent proximal to each of the first client site and proximal to thesecond client site; wherein: communications between the first clientsite's client site network component and associated network servercomponent are bonded or aggregated, then terminated by the networkserver component associated with the first client site's client sitenetwork component, and passed to the high performing network; datatraffic is received by the network server component associated with thesecond client site and transferred on a bonded or aggregated connectionbetween the network server component associated with the second clientsite and the client site network component associated with the secondclient site.
 11. The network system of claim 1, wherein thebonding/aggregation computer system includes a network aggregationdevice that: (A) configures a plurality of dissimilar networkconnections or network connections provided by a plurality of diversenetwork carriers (“diverse network connections”) as one or moreaggregated groups, each aggregated group creating an aggregated networkconnection that is a logical connection of the plurality of diverseconnections; and (B) routes and handles bi-directional transmissionsover the aggregated network connection; wherein two or more of thediverse network connections have dissimilar network characteristicsincluding variable path bidirectional transfer rates and latencies;wherein the logical connection is utilizable for a transfer ofcommunication traffic bidirectionally on any of the diverse networkconnections without any configuration for the dissimilar networkconnections or by the diverse network carriers; and wherein the networkaggregation engine includes or is linked to a network aggregation policydatabase that includes one or more network aggregation policies forconfiguring the aggregated groups within accepted tolerances so as toconfigure and maintain the aggregated network connection so that thelogical connection has a total communication traffic throughput that isa sum of available communication traffic throughputs of the aggregatedgroup of diverse network connections.
 12. A computer implemented methodfor improving network communication performance between at least twosites, where the two sites are at a distance from one another that issuch that would usually require long haul network communication,comprising: (a) using a client site network component associated with afirst client site to connect to a proximal network server component, thenetwork server component being connected to an access point to a highperforming network, forming thereby a network overlay that provides abonded or aggregated connections for carrying data packets; (b) thenetwork server component terminating the bonded or aggregatedconnection; and (c) the network server component transferring the datapackets to the high performing network, for delivery to a second clientsite, while maintaining management of data traffic so as to provide amanaged network path that incorporates at least the bonded/aggregatedconnection and at least one network path carried over the highperforming network, thereby reducing long haul effects.
 13. The methodof claim 12, comprising receiving the data traffic at the second clientsite.
 14. The method of claim 13, comprising maintaining management ofdata traffic so as to provide a managed network path that includes thebonded or aggregated connection and one or more network paths of thehigh performing network.
 15. The method of claim 12, comprisingreceiving the data traffic at a network server component associated withthe second client site, the network server component initiating a bondedor aggregated connection to a client site network component associatedwith the second client site.
 16. The method of claim 12, wherein aplurality of network server components form a Point-of-Presence, and theclient site network component selects one or more of the network servercomponents of the Point-of-Presence for establishing a network overlayso as to improve network performance.